The Many Benefits of Annotator Rationales for Relevance Judgments
نویسندگان
چکیده
When collecting subjective human ratings of items, it can be difficult to measure and enforce data quality due to task subjectivity and lack of insight into how judges arrive at each rating decision. To address this, we propose requiring judges to provide a specific type of rationale underlying each rating decision. We evaluate this approach in the domain of Information Retrieval, where human judges rate the relevance of Webpages. Costbenefit analysis over 10,000 judgments collected on Mechanical Turk suggests a win-win: experienced crowd workers provide rationales with no increase in task completion time while providing further benefits, including more reliable judgments and greater transparency1.
منابع مشابه
Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments
When collecting subjective human ratings of items, it can be difficult to measure and enforce data quality due to task subjectivity and lack of insight into how judges’ arrive at each rating decision. To address this, we propose requiring judges to provide a specific type of rationale underlying each rating decision. We evaluate this approach in the domain of Information Retrieval, where human ...
متن کاملModeling Annotators: A Generative Approach to Learning from Annotator Rationales
A human annotator can provide hints to a machine learner by highlighting contextual “rationales” for each of his or her annotations (Zaidan et al., 2007). How can one exploit this side information to better learn the desired parameters θ? We present a generative model of how a given annotator, knowing the true θ, stochastically chooses rationales. Thus, observing the rationales helps us infer t...
متن کاملImproving Gender Prediction of Social Media Users via Weighted Annotator Rationales
This paper proposes and contrastively evaluates several novel approaches to utilizing annotator rationales to improve the prediction of user gender in social media for English and Spanish. Our methods outperform state-of-the-art systems for Twitter gender prediction, and yield up to 28% error reduction relative to an otherwise identical system and training data without the use of annotator rati...
متن کاملAutomatically Generating Annotator Rationales to Improve Sentiment Classification
One of the central challenges in sentimentbased text categorization is that not every portion of a document is equally informative for inferring the overall sentiment of the document. Previous research has shown that enriching the sentiment labels with human annotators’ “rationales” can produce substantial improvements in categorization performance (Zaidan et al., 2007). We explore methods to a...
متن کاملMachine Learning with Annotator Rationales to Reduce Annotation Cost
We review two novel methods for text categorization, based on a new framework that utilizes richer annotations that we call annotator rationales. A human annotator provides hints to a machine learner by highlighting contextual “rationales” in support of each of his or her annotations. We have collected such rationales, in the form of substrings, for an existing document sentiment classification...
متن کامل